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What Is The Nation’s Report Card™? 



The Nations Report Card™, the National Assessment of 
Educational Progress (NAEP), is a nationally representative 
and continuing assessment of what Americas students know 
and can do in various subject areas. For over three decades, 
assessments have been conducted periodically in reading, 
mathematics, science, writing, history, geography, and other 
subjects. 

NAEP is a congressionally mandated project of the 
National Center for Education Statistics within the 
Institute of Education Sciences of the U.S. Department 
of Education. The Commissioner of Education Statistics 
is responsible, by law, for carrying out the NAEP project 
through competitive awards to qualified organizations. 

By making objective information on student performance 
available to policymakers at the national, state, and local 
levels, NAEP is an integral part of our nations evaluation 
of the condition and progress of education. Only informa- 
tion related to academic achievement and relevant variables 



is collected under this program. The privacy of individual 
students and their families is protected, and the identities of 
participating schools are not released. 

In 1988, Congress established the National Assessment 
Governing Board (NAGB) to oversee and set policy for 
NAEP The Board is responsible for selecting the subject 
areas to be assessed; setting appropriate student achievement 
levels; developing assessment objectives and test specifica- 
tions; developing a process for the review of the assessment; 
designing the assessment methodology; developing 
guidelines for reporting and disseminating NAEP results; 
developing standards and procedures for interstate, regional, 
and national comparisons; determining the appropriateness 
of all assessment items and ensuring the assessment items 
are free from bias and are secular, neutral, and nonideo- 
logical; taking actions to improve the form, content, use, 
and reporting of results of the National Assessment; and 
planning and executing the initial public release of NAEP 
reports. 
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Executive Summary 



The goal of the study was to examine differences in 
mean National Assessment of Educational Progress 
(NAEP) reading and mathematics scores between pub- 
lic and private schools when selected characteristics 
of students and / or schools were taken into account. 
Among the student characteristics considered were gen- 
der, race/ethnicity, disability status, and identification 
as an English language learner. Among the school char- 
acteristics considered were school size and location, and 
composition of the student body and of the teaching 
staff. In particular, if the student populations enrolled 
in the two types of schools differed systematically with 
respect to background characteristics related to achieve- 
ment, then those differences would be confounded with 
straightforward comparisons between school types. 

The present report examined results from the 2003 
NAEP assessments in reading and mathematics for 
grades 4 and 8. NAEP draws nationally representative 
samples of schools and students. In 2003, over 6,900 
public schools and over 530 private schools participated 
in the grade 4 assessments. Over 5,500 public schools 
and over 550 private schools participated in the grade 8 
assessments. 

Elierarchical linear models (HLMs) were employed to 
carry out the desired adjustments. EILMs were a natural 
choice because they accommodate the nested structure 
of the data (i.e., students clustered within schools) 
and facilitate the inclusion of variables derived from 
student and school characteristics. In this study, the 
focal parameter was the mean difference between mean 
NAEP scores for two populations of schools. (This 
difference was not identical to the difference in mean 
scores between the two student populations, though the 
discrepancy was typically small.) HLMs were used to 
compare all private schools to all public schools, as well 
as to compare, separately, certain categories of private 
schools (i.e., those for which sample sizes were suffi- 
cient to report reliable estimates) to all public schools. 
Statistical significance was determined at the .05 level 
using t tests on model results. 



Results From Grade 4 

Reading 

In the first set of analyses, all private schools were 
compared to all public schools. The average private 
school mean reading score was 14.7 points higher 
than the average public school mean reading score, 
corresponding to an effect size of .41 (the ratio of the 
absolute value of the estimated difference to the stan- 
dard deviation of the NAEP fourth-grade reading score 
distribution). After adjusting for selected student char- 
acteristics, the difference in means was near zero and 
not significant. In the second set of analyses, Catholic 
schools and Lutheran schools were each compared to 
all public schools. The results, both with and without 
adjustments, were similar to the corresponding results 
for all private schools. 

Mathematics 

In the first set of analyses, all private schools were again 
compared to all public schools. The average private 
school mean mathematics score was 7.8 points higher 
than the average public school mean mathematics score, 
corresponding to an effect size of .29. After adjusting 
for selected student characteristics, the difference in 
means was -4.5 and significantly different from zero. 
(Note that a negative difference implies that the aver- 
age school mean was higher for public schools.) In the 
second set, Catholic schools and Lutheran schools were 
each compared to all public schools. The results, both 
with and without adjustments, were similar to the cor- 
responding results for all private schools. 

Results From Grade 8 

Reading 

In the first set of analyses, all private schools were com- 
pared to all public schools. The average private school 
mean reading score was 18.1 points higher than the 
average public school mean reading score, correspond- 
ing to an effect size of .58. After adjusting for selected 
student characteristics, the difference in means was 
7.3 points and significantly different from zero. In 
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the second set, Catholic, Lutheran, and Conservative 
Christian schools were each compared to all public 
schools. The results, both with and without adjust- 
ments, were generally similar to the corresponding 
results for all private schools. The only exception was 
that the average difference in adjusted school mean 
scores between Conservative Christian schools and all 
public schools was not significantly different from zero. 

Mathematics 

In the first set of analyses, all private schools were 
again compared to all public schools. The average pri- 
vate school mean mathematics score was 12.3 points 
higher than the average public school mean mathemat- 
ics score, corresponding to an effect size of .38. After 
adjusting for selected student characteristics, the differ- 
ence in means was nearly zero and not significant. In 
the second set, Catholic, Lutheran, and Conservative 
Christian schools were each compared to all public 
schools. While the results for Catholic schools, both 
with and without adjustments, were very similar to the 
corresponding results for all private schools, the results 
for the other two types differed. 

The initial difference between Lutheran schools and 
all public schools was substantially larger (19.5 points) 
than was the case for all private schools. The aver- 
age difference in adjusted mean mathematics scores 
between the two types of schools was 4.9 points and 
significantly different from zero. On the other hand, 
the initial difference between Conservative Christian 
schools and all public schools was substantially smaller 
(5.1 points) and not significant. The average differ- 
ence in adjusted school means between Conservative 
Christian schools and all public schools was -7.6 points 
(i.e., a higher average school mean for public schools) 
and was significantly different from zero. 



Comparison of Results for Grade 4 
and Grade 8 

Overall, there were many similarities in the results 
for the two grades. In both reading and mathematics, 
analyses employing unadjusted NAEP scores indicated 
that the average private school mean score was higher 
than the average public school mean score, and the dif- 
ference was statistically significant. Including selected 
student characteristics in the model, however, resulted 
in a substantial reduction in the difference in all four 
analyses. The reduction varied from 11 to 15 score 
points. For grade 4 reading and grade 8 mathematics, 
the average difference in adjusted school mean scores 
was no longer significant. For grade 4 mathematics, 
the difference was significant, and the adjusted school 
mean was higher for public schools. Only for grade 8 
reading was the difference still significant with a higher 
school mean for private schools. For all four analyses, 
with student characteristics such as gender and race/ 
ethnicity incorporated in the model, the inclusion of 
school characteristics (e.g., teacher experience, type of 
school location, school size) had little impact on the 
estimate of the average difference between the two 
types of schools. 

Variance decompositions yielded similar results for 
the four grade-subject combinations. Most of the total 
variance was due to heterogeneity among students 
within schools rather than heterogeneity among school 
mean scores. The combination of selected student and 
school characteristics accounted for about one-third of 
the total variance for grade 4 and about two-fifths of 
the total variance for grade 8. 
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Cautions in Interpretation 

When interpreting the results from any of these analy- 
ses, it should be borne in mind that private schools 
constitute a heterogeneous category and may differ 
from one another as much as they differ from public 
schools. Public schools also constitute a heterogeneous 
category. Consequently, an overall comparison of the 
two types of schools is of modest utility. The more 
focused comparisons conducted as part of this study 
may be of greater value. However, interpretations of 
the results should take into account the variability due 
to the relatively small sizes of the samples drawn from 
each category of private school, as well as the possible 
bias introduced by the differential participation rates 
across private school categories. 

There are a number of other caveats. First, the con- 
clusions pertain to national estimates. Results based 
on a survey of schools in a particular jurisdiction may 
differ. Second, the data are obtained from an observa- 
tional study rather than a randomized experiment, so 
the estimated effects should not be interpreted in terms 
of causal relationships. In particular, private schools 
are “schools of choice.” Without further information, 
such as measures of prior achievement, there is no way 
to determine how patterns of self-selection may have 
affected the estimates presented. That is, the estimates 
of the average difference in school mean scores are 
confounded with average differences in the student 
populations, which are not fully captured by the select- 
ed student characteristics employed in this analysis. 



Summary 

In grades 4 and 8 for both reading and mathematics, 
students in private schools achieved at higher levels 
than students in public schools. The average difference 
in school means ranged from almost 8 points for grade 4 
mathematics, to about 18 points for grade 8 reading. 
The average differences were all statistically significant. 
Adjusting the comparisons for student characteristics 
resulted in reductions in all four average differences 
of approximately 11 to 14 points. Based on adjusted 
school means, the average for public schools was sig- 
nificantly higher than the average for private schools 
for grade 4 mathematics, while the average for private 
schools was significantly higher than the average for 
public schools for grade 8 reading. The average differ- 
ences in adjusted school means for both grade 4 reading 
and grade 8 mathematics were not significantly differ- 
ent from zero. 

Comparisons were also carried out with subsets of 
private schools categorized by sectarian affiliation. 

After adjusting for student characteristics, raw score 
average differences were reduced by about 11 to 15 
points. In grade 4, Catholic and Lutheran schools were 
each compared to public schools. For both reading 
and mathematics, the results were generally similar to 
those based on all private schools. In grade 8, Catholic, 
Lutheran, and Conservative Christian schools were each 
compared to public schools. For Catholic and Lutheran 
schools for both reading and mathematics, the results 
were again similar to those based on all private schools. 
For Conservative Christian schools, the average 
adjusted school mean in reading was not significantly 
different from that of public schools. In mathemat- 
ics, the average adjusted school mean for Conservative 
Christian schools was significantly lower than that of 
public schools. 
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Chapter 1 



Introduction 

There is an extensive research literature that treats 
questions related to comparisons of public and private 
schools. An excellent review is provided by McEwan 
(2000), who argues that, with a few exceptions, there is 
generally insufficient evidence to reach strong conclu- 
sions with regard to such comparisons. Methodological 
difficulties found in this literature include the size and 
nature of the available samples of schools and students 
(e.g., small sample sizes, self-selection into public or 
private schools), as well as key student, family, school, 
and community variables that remain unmeasured 
but may be associated with both public versus private 
school attendance and student achievement. 

A previous National Assessment of Educational 
Progress (NAEP) report on the achievement of students 
in private schools (Perie, Vanneman, and Goldstein 
2005) compared the NAEP reading and mathemat- 
ics performance of fourth-, eighth-, and (for some 
findings) twelfth-grade students attending public and 
private schools. Results were also presented disaggre- 
gated by type of private school. In general, the average 
scores in reading and mathematics of students in pri- 
vate schools were found to be higher than those of 
students in public schools. 

A natural question is whether these differences can 
be accounted for by differences in the populations of 
students attending the various kinds of schools. The 
previous NAEP report also presented results disag- 
gregated both by school type and by a single student 
characteristic such as race/ethnicity, gender, or student- 
reported parents’ highest level of education. Generally, 
the differences between public and private school stu- 
dent performance were diminished somewhat in these 
disaggregated analyses, but the average scores of private 
school students remained higher than those of compa- 
rable public school students; for example, students of 



the same race/ethnicity. The further question remains, 
however, as to whether these observed differences 
would persist if the comparisons were made between 
subgroups of private school and public school students 
who were similar with respect to several characteristics 
at once, for example, race/ ethnicity, gender, and par- 
ents’ education. 

More complex analysis techniques that allow mul- 
tiple covariates (e.g., race/ethnicity, eligibility for 
free/ reduced-price school lunch) to be statistically con- 
trolled within the same analysis can be used to address 
this question. The present report adds to the previously 
reported findings by employing a particular type of 
analysis, called hierarchical linear modeling, the essen- 
tials of which are described below. The report examines 
reading and mathematics data for both grades 4 and 8 
from the 2003 NAEP administration. Use of hierarchi- 
cal linear modeling not only facilitates the examination 
of multiple covariates simultaneously, but also takes 
account of the clustering of students within schools. 

One recent study (Lubienski and Lubienski 2006) 
also employed hierarchical linear models to examine 
public versus private students’ performance on the 
2003 NAEP assessment. The general conclusion 
from that study was “ . . . that demographic differ- 
ences between students in public and private schools 
more than account for the relatively high raw scores of 
private schools. Indeed, after controlling for these dif- 
ferences, the presumably advantageous private school 
effect’ disappears, and even reverses in most cases” 
(Lubienski and Lubienski 2006, p. 3). The analyses 
conducted for the current report differ from those 
reported by Lubienski and Lubienski in two main 
respects. First, they focused exclusively on mathemat- 
ics in grades 4 and 8, while the current report has a 
dual focus on reading and mathematics. Second, they 
divided public school students into those attending 
charter schools and those attending public noncharter 
schools — an analysis that is not included in the current 
report. 
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CHAPTER 1 



Overview of NAEP 

Since 1971, NAEP has been an ongoing, nationally 
representative indicator of what students know and can 
do in a variety of academic subjects. Over the years, 
NAEP has measured student achievement in many 
subjects, including reading, mathematics, science, writ- 
ing, U.S. history, geography, civics, and the arts. NAEP 
is administered by the National Center for Education 
Statistics (NCES), within the U.S. Department of 
Education’s Institute of Education Sciences, and is 
overseen by the National Assessment Governing Board 
(NAGB). 

In 2003, NAEP assessments in reading and math- 
ematics were conducted at grades 4 and 8. The content 
of each assessment was determined by subject- area 
frameworks developed by NAGB with input from a 
broad spectrum of educators, parents, and members of 
the general public. The complete frameworks for the 
NAEP reading and mathematics assessments are avail- 
able on the NAGB website (www.nagb.org/pubs/pubs. 
html) . Additional information about the design of the 
2003 assessments is provided in appendix A. 

NAEP is not designed to provide scores for indi- 
vidual students and schools; instead, it provides results 
regarding subject-matter achievement, instructional 
experiences, and school environment for populations of 
students and groups of students in those populations. 
Through the use of complex item-sampling designs 
that present each participating student with only a por- 
tion of the total assessment, NAEP is able to produce 
accurate estimates of the performance of students in 
the nation, while minimizing the time burden on any 
individual student or school. However, NAEP is not 
designed to provide information that directly addresses 
the issue of how public and private schools contributed 
to a student's education. 



Private schools have a long history in American 
education. About 10 percent of the entire school popu- 
lation, almost 5.3 million American students, attended 
private schools during the 2001-2002 school year 
(Broughman and Puch 2004). NAEP has reported on 
the performance of private school students since 1977. 
A recent report drew on results from NAEP assess- 
ments administered from 2000 through 2005 (Perie, 
Vanneman, and Goldstein 2005). In view of the inter- 
est in the question of whether different types of schools 
are equally effective in helping all students to learn, 
NAGB, which sets policy for NAEP, asked NCES to 
conduct a special study to supplement the standard 
report that documents student achievement in pri- 
vate and public schools. The results presented in this 
report are based on additional analyses, using statistical 
modeling techniques that take demographic and other 
contextual differences into account in estimating dif- 
ferences in how attending public versus private schools 
relates to students’ performance on NAEP. 

School and Student Samples 

In 2003, over 6,900 public schools and over 530 pri- 
vate schools participated in the NAEP assessments in 
reading and mathematics at grade 4. Over 5,500 public 
schools and over 550 private schools participated at 
grade 8. Within each participating school, a random 
sample of students participated in either the reading 
or mathematics assessment — about one-half partici- 
pated in reading and about one-half participated in 
mathematics in public schools, and a higher proportion 
participated in reading than in mathematics in private 
schools. Table 1-1 displays the number of students 
sampled for the reading and mathematics assessments at 
each grade by type of school. The number of students 
sampled is likely to differ slightly from the number of 
students who were actually assessed as a result of exclu- 
sion and absentees. (See appendix A for additional 
information on school and student participation rates 
by type of school.) Table 1-2 shows the number of 
participating schools and students' average scores in 
reading and mathematics by type of school. 
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Every effort is made to ensure that all sampled stu- 
dents who are capable of participating in the assessment 
are assessed. Sampled students who are identified by 
the school as students with disabilities or as English 
language learners are included in the assessment 
unless they do not meet criteria for inclusion estab- 
lished by NAEP. 1 Such students may be assessed with 
accommodations allowed by NAEP. Five percent of 
the fourth- and eighth-grade public school students 
sampled in 2003 were assessed with accommoda- 
tions in reading, and 7 to 8 percent were assessed with 



accommodations in mathematics. Of the fourth- and 
eighth-grade private school students sampled, 2 percent 
were assessed with accommodations in each subject. At 
grade 4, public schools in the national sample excluded 
4 percent of students in mathematics and 6 percent in 
reading in 2003. At grade 8, the exclusion rates in public 
schools were 4 percent in mathematics and 5 percent 
in reading in 2003. The comparable exclusion rates in 
private schools were 1 percent or less in both subjects at 
both grades. 



Table 1-1. Student sample size for NAEP reading and math- 
ematics assessments, by type of school, grades 4 
and 8: 2003 





Student sample size by subject 


Type of school 


Reading 


Mathematics 


Grade 4 






Public 


191,400 


191,400 


Private 


7,500 


4,700 


Catholic 


3,700 


2,300 


Lutheran 


900 


600 


Conservative Christian 


1,000 


700 


Grade 8 






Public 


155,000 


153,500 


Private 


8,300 


5,100 


Catholic 


4,000 


2,500 


Lutheran 


1,000 


600 


Conservative Christian 


1,100 


700 



NOTE: The number of students sampled for the combined private total includes students 
in the “other private school” category, which is not listed separately in the table. Sample 
sizes are rounded to the nearest hundred. 



SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center 
for Education Statistics, National Assessment of Educational Progress (NAEP), 2003 
Reading and Mathematics Private School Study. 



Table 1-2. Number of schools and students’ average NAEP 

reading and mathematics scores, by type of school, 
grades 4 and 8: 2003 



Type of school 


Reading 


Mathematics 


Number 

of Average 

schools score 


Number 

of Average 

schools score 


Grade 4 






Public 


6,908 216(0.3) 


6,914 234(0.2) 


Private 


542 235(0.8) 


539 244(0.7) 


Catholic 


215 235(1.0) 


216 244(0.8) 


Lutheran 


90 232(1.9) 


88 245(1.5) 


Conservative Christian 


79 * 


00 

-H- 


Grade 8 






Public 


5,531 261 (0.2) 


5,527 276(0.3) 


Private 


568 282(0.7) 


558 292 (1.2) 


Catholic 


224 281(0.9) 


224 289(1.4) 


Lutheran 


101 281(1.6) 


96 296(1.6) 


Conservative Christian 


92 276(1.5) 


90 286(2.6) 



$ Reporting standards not met. 

NOTE: Schools participating in the combined private school category include those in 
the “other private school” category, which is not listed separately in the table. Standard 
errors of the average scores appear in parentheses. 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center 
for Education Statistics, National Assessment of Educational Progress (NAEP), 2003 
Reading and Mathematics Private School Study. 



1 The percentages of students identified as students with disabilities or English language learners were 19 to 22 percent in public schools and 3 to 4 percent in private 
schools (see tables A-7 and A-8 in appendix A). 
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CHAPTER 1 



Cautions in Interpretation 

NAEP data are collected as part of an observational 
study rather than as a randomized experiment. In 
particular, families choose to enroll their children in 
private schools, and it is possible that there are system- 
atic differences between those families and the general 
population of families that are not captured by the 
student characteristics available for analysis. If such 
differences are correlated with student achievement, 
then the estimated average difference in achievement 
between public school students and private school 
students (even after adjusting for observed student 
characteristics) will be confounded to some degree with 
the unobserved differences between the families of the 
children in the two school types. This is usually termed 
“selection bias.” Although this study employs a power- 
ful statistical tool, hierarchical linear models, it cannot 
fully compensate for the lack of relevant data. The 
implication is that the estimated effects obtained should 
not be interpreted in terms of causal relationships. 

Relevant unobserved variables may include (but are 
not limited to) the following: 

1 . A prior measure of achievement, which is often 
employed as a covariate in such studies. Apparent 
differences in average achievement between public 
school students and private school students may 
simply reflect average differences in achievement 
between their respective student populations (at 
entry into the fourth or eighth grade) that are 
not adequately captured by observed student 
characteristics. 

2. The possible attraction of parents to private 
schools because they felt that their children were 
not well served by public schools. The interpreta- 
tion of the results is further clouded by the fact 
that, typically, students in the private school sam- 
ple will have spent different amounts of time in 
the private school system. 

3. The extent to which parents provide support and 
encouragement for academic achievement. 

Inasmuch as NAEP draws samples of schools and 
students, estimates of the differences in achievement 



between school types are subject to uncertainty. In 
particular, the number of private schools in the sample 
is an order of magnitude smaller than the number of 
public schools. Consequently, the (estimated) standard 
errors of the difference estimates will tend to be higher 
than one might expect, given the total number of 
schools in the sample, because they are strongly influ- 
enced by the size of the smaller sample. 

It should be borne in mind that in both grades 4 
and 8 the school-level response rates for each type of 
private school are lower than those for public schools. 
(See tables A- 5 and A-6 in appendix A.) In both grades, 
Conservative Christian schools have the lowest response 
rates. This self-selection may introduce bias into the 
reported comparisons, and this bias would persist 
even after adjusting for observed student characteris- 
tics. Note also that the estimated standard errors that 
accompany each statistic do not reflect this bias. 

Finally, public and private schools differ substantially 
in the weighted percentages of students sampled who 
are classified as having disabilities or being English lan- 
guage learners. (See tables A-7 and A- 8 in appendix A.) 
Since classified students tend to perform more poorly 
on average than other students (Donahue, Daane, 
and Grigg 2003; Braswell, Daane, and Grigg 2003), 
such differences in the composition of their student 
populations contribute to the observed differences in 
achievement between school types and should be taken 
into account in their interpretation. 

Overview of Study Design 
and Application of 
Hierarchical Linear Modeling 

Elierarchical linear modeling (FILM) is a class of 
techniques for analyzing data having a hierarchical or 
nested structure. For example, a database may consist of 
students who are nested within the schools they attend. 
Analyzing such data structures poses special problems. 
Conventional regression techniques either treat the 
school as the unit of analysis (ignoring the variation 
among students within schools) or treat the student 
as the unit of analysis (ignoring the nesting within 
schools). Neither approach is satisfactory. 
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In the former case, valuable information is lost, and 
the fitted school-level model can misrepresent the 
relationships among variables at the student level. In 
the latter case, it is assumed that if the model is cor- 
rectly specified, then all the observations (e.g., student 
outcomes) are independent of one another. However, 
students attending the same school share many com- 
mon, educationally relevant experiences that affect 
academic performance. As a result, scores on academic 
measures for students in the same school will not be 
independent, even after adjusting for student charac- 
teristics. Violation of the independence assumption 
means that, typically, estimates of standard errors of 
means and regression weights related to academic 
performance will be biased. Such bias, in turn, leads 
to situations in which statements of significance can 
occur too often or not often enough; that is, the actu- 
al Type I or Type II error rates can be quite different 
from the nominal ones. 2 

With HLM, on the other hand, the nested structure 
is represented explicitly in a multilevel model, with 
different variances assumed for each level. This amelio- 
rates the above-mentioned problems with single-level 
models. Moreover, it is possible to postulate a separate 
student-level regression for each school. Both student 
and school characteristics can be included, and standard 
errors of means and regression coefficients can be esti- 
mated without bias. Consequently, the corresponding 
significance tests have the proper Type I and Type II 
error rates. For further discussion, see Raudenbush and 
Bryk, chapter 1 (2002). 

Hierarchical linear models are very flexible. They 
consist of two or more sets of linear regression equa- 
tions that can incorporate explanatory variables at each 
level of the data structure. In the example above, at the 
lower level (level 1) there is a regression equation for 
each school relating a students outcome to one or more 
student characteristics (e.g., gender, race, socioeco- 
nomic status). The relationship between test scores and 
students’ characteristics, represented by a set of regres- 
sion coefficients, can differ from one school to another. 



At the higher level (level 2), each school’s set of regres- 
sion coefficients is predicted by one or more school 
characteristics (e.g., school type, school size, racial com- 
position). 

An analysis based on HLM yields a decomposition 
of the total variance into a between-student, within- 
school component and a between-school component. 

In addition, the output of the level 1 regression tells 
how much of the variation in test scores between stu- 
dents within schools (i.e., the first component) can be 
accounted for by differences in student characteristics. 
Similarly, the output of a particular level 2 regression 
tells how much of the variation in school means, or 
adjusted school means (i.e., the second component), 
can be accounted for by differences in school charac- 
teristics such as the public/private designation. Because 
the NAEP database conforms to a hierarchical struc- 
ture — students nested within schools — HLM is well 
suited for carrying out an investigation that can help to 
elucidate the differences in achievement between public 
and private schools. 

Ideally, to ascertain the difference between the two 
types of schools, an experiment would be conducted in 
which students are assigned (by an appropriate random 
mechanism) to either public or private schools. With a 
sufficiently large sample, such a procedure would guar- 
antee that, on average, there are no initial differences 
between students attending public or private schools, 
and would facilitate a more controlled comparison of 
the two types of schools. However, students are not 
randomly assigned to schools; families choose to seek 
admission for their children to private schools. Thus, 
it is possible that students enrolled in the two types of 
schools differ on key characteristics that are associated 
with achievement. To the extent that is true, estimates 
of the average difference in achievement between 
school types will be confounded with initial differences 
between their student populations. This is of special 
concern if measures of prior academic achievement are 
unavailable, as is the case here. 



2 The Type I error rate is the probability that a statistical test will (incorrectly) reject a null hypothesis of no difference when the null hypothesis is true. The Type I 
error rate is set in advance of the analysis, and .05 is a typical value. The Type II error rate is the probability that a statistical test will (incorrectly) accept a null 
hypothesis when the null hypothesis is false. The Type II error rate is determined by the Type I error rate, the statistical test used, and the extent of the departure 
from the null hypothesis. 



